Imaging device, method for controlling imaging device, and control program

ABSTRACT

A projection controller controls an infrared projector to selectively and sequentially project first to third infrared lights. An imaging unit images an object in a state where the first to third infrared lights are projected so as to generate first to third frames. An image processing unit synthesizes the first to third frames to generate a frame of an image signal. The projection controller sets an interval between a first timing and a second timing shorter than an interval between the first timing and a third timing. The first timing is the middle point of the period in which the second infrared light is projected. The second timing is the middle point of the period in which the first or third infrared light is projected. The third timing is the middle point of the one frame period of the first or third frame.

CROSS REFERENCE TO RELATED APPLICATION

This application is a Continuation of PCT Application No.PCT/JP2015/072892, filed on Aug. 13, 2015, and claims the priority ofJapanese Patent Application No. 2014-178000, filed on Sep. 2, 2014, theentire contents of both of which are incorporated herein by reference.

BACKGROUND

The present disclosure relates to an imaging device, a method forcontrolling an imaging device, and a control program.

There is known a method for imaging an object under the condition thatalmost no visible light is available, such as during nighttime, byradiating infrared light onto the object from an infrared projector andimaging infrared light reflected by the object. This imaging method iseffective in a case where lighting fixtures for radiating visible lightcannot be used.

However, since an image obtained by imaging the object by this method isa monochromatic image, it is difficult to identify the object from themonochromatic image depending on circumstances. If a color image can becaptured even under the condition that no visible light is available,the performance of identifying the object can be improved. For example,it is expected that surveillance cameras can capture color images underthe condition that no visible light is available in order to improveperformance for identifying objects.

Japanese Unexamined Patent Application Publication No. 2011-050049(Patent Document 1) describes an imaging device capable of capturingcolor images under the condition that no visible light is available. Theimaging device described in Patent Document 1 uses an infraredprojector. Incorporating the technique described in Patent Document 1into a surveillance camera can capture a color image of an object so asto improve the identification of the object.

SUMMARY

The imaging device, using the infrared projector, may cause variationsin color when imaging a moving object by radiating infrared light ontothe object.

A first aspect of the embodiments provides an imaging device including:a projection controller configured to control an infrared projector toselectively and sequentially project a first infrared light having afirst wavelength assigned to a first color of red, green, and blue, asecond infrared light having a second wavelength assigned to a secondcolor of red, green, and blue, and a third infrared light having a thirdwavelength assigned to a third color of red, green, and blue; an imagingunit configured to image an object in a state where the first infraredlight is projected in at least part of one frame period so as togenerate a first frame based on a first imaging signal, image the objectin a state where the second infrared light is projected in at least partof the one frame period so as to generate a second frame based on asecond imaging signal, and image the object in a state where the thirdinfrared light is projected in at least part of the one frame period soas to generate a third frame based on a third imaging signal; and animage processing unit configured to synthesize the first to third framesto generate a frame of an image signal, wherein the projectioncontroller sets an interval between a first timing and a second timingshorter than an interval between the first timing and a third timing,the first timing being a middle point of a period in which the secondinfrared light is projected, the second timing being a middle point of aperiod in which the first or third infrared light is projected, and thethird timing being a middle point of the one frame period of the firstor third frame, and controls the infrared projector to project the firstto third infrared lights.

A second aspect of the embodiments provides a method for controlling animaging device, including: a first step of imaging an object by animaging unit in a state where a first infrared light having a firstwavelength assigned to a first color of red, green, and blue isprojected in at least part of one frame period so as to generate a firstframe based on a first imaging signal; a second step, implemented afterthe first step, of imaging the object by the imaging unit in a statewhere a second infrared light having a second wavelength assigned to asecond color of red, green, and blue is projected in at least part ofthe one frame period so as to generate a second frame based on a secondimaging signal; a third step, implemented after the second step, ofimaging the object by the imaging unit in a state where a third infraredlight having a third wavelength assigned to a third color of red, green,and blue is projected in at least part of the one frame period so as togenerate a third frame based on a third imaging signal; and a fourthstep of synthesizing the first to third frames to generate a frame of animage signal, wherein an interval between a first timing and a secondtiming is set shorter than an interval between the first timing and athird timing, the first timing being a middle point of a period in whichthe second infrared light is projected, the second timing being a middlepoint of a period in which the first or third infrared light isprojected, and the third timing being a middle point of the one frameperiod of the first or third frame.

A third aspect of the embodiments provides a control program of animaging device executed by a computer and stored in a non-transitorystorage medium to implement the following steps, including: a first stepof controlling an infrared projector to project a first infrared lighthaving a first wavelength assigned to a first color of red, green, andblue; a second step of imaging an object by an imaging unit in a statewhere the first infrared light is projected in at least part of oneframe period so as to generate a first frame based on a first imagingsignal; a third step, continued from the first step, of controlling theinfrared projector to project a second infrared light having a secondwavelength assigned to a second color of red, green, and blue; a fourthstep of imaging the object by the imaging unit in a state where thesecond infrared light is projected in at least part of the one frameperiod so as to generate a second frame based on a second imagingsignal; a fifth step, continued from the third step, of controlling theinfrared projector to project a third infrared light having a thirdwavelength assigned to a third color of red, green, and blue; a sixthstep of imaging the object by the imaging unit in a state where thethird infrared light is projected in at least part of the one frameperiod so as to generate a third frame based on a third imaging signal;and a seventh step of synthesizing the first to third frames to generatea frame of an image signal, wherein the control program sets an intervalbetween a first timing and a second timing shorter than an intervalbetween the first timing and a third timing, the first timing being amiddle point of a period in which the second infrared light isprojected, the second timing being a middle point of a period in whichthe first or third infrared light is projected, and the third timingbeing a middle point of the one frame period of the first or thirdframe.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an overall configuration of an imagingdevice according to at least one embodiment.

FIG. 2 is a view showing an example of an array of filter elements in acolor filter used in the imaging device according to the embodiment.

FIG. 3 is a characteristic diagram showing spectral sensitivecharacteristics of wavelengths and relative sensitivities of light ofthree primary colors in an imaging unit included in the imaging deviceaccording to the embodiment.

FIG. 4 is a characteristic diagram showing a relationship betweenwavelengths and relative detection rates when multiplying, by a lightreceiving sensitivity of silicon, a reflectance of light of each primarycolor obtained from a particular substance.

FIG. 5 is a block diagram showing a specific configuration example of apre-signal processing unit 52 shown in FIG. 1.

FIG. 6 is a view showing a relationship between exposures and frames ofimage signals when the imaging device according to the embodiment isoperating in a normal mode.

FIG. 7 is a view for describing demosaicing when the imaging deviceaccording to the embodiment is operating in the normal mode.

FIG. 8 is a view showing a relationship between exposures and frames ofimage signals when the imaging device according to the embodiment isoperating in an intermediate mode and in a night-vision mode.

FIG. 9 is a view for describing pre-signal processing when the imagingdevice according to the embodiment is operating in a first intermediatemode.

FIG. 10 is a view for describing demosaicing when the imaging deviceaccording to the embodiment is operating in the first intermediate mode.

FIG. 11 is a view for describing pre-signal processing when the imagingdevice according to the embodiment is operating in a second intermediatemode.

FIG. 12 is a view for describing demosaicing when the imaging deviceaccording to the embodiment is operating in the second intermediatemode.

FIG. 13 is a view for describing processing of adding surrounding pixelswhen the imaging device according to the embodiment is operating in thenight-vision mode.

FIG. 14 is a view showing frames on which the processing of adding thesurrounding pixels is performed.

FIG. 15 is a view for describing pre-signal processing when the imagingdevice according to the embodiment is operating in a first night-visionmode.

FIG. 16 is a view for describing demosaicing when the imaging deviceaccording to the embodiment is operating in the first night-vision mode.

FIG. 17 is a view for describing pre-signal processing when the imagingdevice according to the embodiment is operating in a second night-visionmode.

FIG. 18 is a view for describing demosaicing when the imaging deviceaccording to the embodiment is operating in the second night-visionmode.

FIG. 19 is a view for describing an example of a mode switch in theimaging device according to the embodiment.

FIG. 20 is a view showing conditions of the respective members when theimaging device according to the embodiment is set to the respectivemodes.

FIG. 21 is a partial block diagram showing a first modified example ofthe imaging device according to the embodiment.

FIG. 22 is a partial block diagram showing a second modified example ofthe imaging device according to the embodiment.

FIG. 23 is a partial block diagram showing a third modified example ofthe imaging device according to the embodiment.

FIG. 24 is a flowchart showing an image signal processing method.

FIG. 25 is a flowchart showing specific processing steps in the normalmode shown in step S3 of FIG. 24.

FIG. 26 is a flowchart showing specific processing steps in theintermediate mode shown in step S4 of FIG. 24.

FIG. 27 is a flowchart showing specific processing steps in thenight-vision mode shown in step S5 of FIG. 24.

FIG. 28 is a flowchart showing processing steps executed by a computerdirected by an image signal processing program.

FIG. 29 is a timing chart schematically showing a method for controllingthe imaging device when the imaging device generates a frame of an imagesignal while taking no account of variations in color.

FIG. 30 is a timing chart schematically showing a first example of amethod for controlling the imaging device that can minimize variationsin color when generating a frame of an image signal.

FIG. 31 is a timing chart schematically showing a second example of themethod for controlling the imaging device that can minimize variationsin color when generating a frame of an image signal.

FIG. 32 is a timing chart schematically showing a third example of themethod for controlling the imaging device that can minimize variationsin color when generating a frame of an image signal.

DETAILED DESCRIPTION

Hereinafter, an imaging device, a method for controlling an imagingdevice, and a control program according to the embodiment will bedescribed with reference to appended drawings.

<Configuration of Imaging Device>

First, the entire configuration of the imaging device according to theembodiment is described below with reference to FIG. 1. The imagingdevice according to the embodiment shown in FIG. 1 is capable ofcapturing images in three modes including a normal mode suitable forimaging in a state where sufficient visible light is present such asduring the day, a night-vision mode suitable for imaging in a statewhere almost no visible light is present such as at night, and anintermediate mode suitable for imaging in a state where visible light isslightly present.

The intermediate mode is a first infrared light projecting mode forimaging while projecting infrared light under the condition that theamount of visible light is small. The night-vision mode is a secondinfrared light projecting mode for imaging while projecting infraredlight under the condition that the amount of visible light is smaller(almost no visible light is present).

The imaging device may include either the intermediate mode or thenight-vision mode. The imaging device does not necessarily include thenormal mode. The imaging device is only required to include an infraredlight projecting mode for imaging while projecting infrared light.

As shown in FIG. 1, a light indicated by the dash-dotted line reflectedby an object is collected by an optical lens 1. Visible light enters theoptical lens 1 under the condition that visible light is presentsufficiently, and infrared light emitted from an infrared projector 9described below and reflected by the object enters the optical lens 1under the condition that almost no visible light is present.

In the state where visible light is slightly present, mixed lightincluding both the visible light and the infrared light emitted from theinfrared projector 9 and reflected by the object, enters the opticallens 1.

Although FIG. 1 shows only one optical lens 1 for reasons ofsimplification, the imaging device actually includes a plurality ofoptical lenses.

An optical filter 2 is interposed between the optical lens 1 and animaging unit 3. The optical filter 2 includes two members; an infraredcut filter 21 and a dummy glass 22. The optical filter 2 is driven by adrive unit 8 in a manner such that the infrared cut filter 21 isinserted between the optical lens 1 and the imaging unit 3 or such thatthe dummy glass 22 is inserted between the optical lens 1 and theimaging unit 3.

The imaging unit 3 includes an imaging element 31 in which a pluralityof light receiving elements (pixels) are arranged in both the horizontaldirection and the vertical direction, and a color filter 32 in whichfilter elements of red (R), green (G), or blue (B) corresponding to therespective light receiving elements are arranged. The imaging element 31may be either a charge coupled device (CCD) or a complementary metaloxide semiconductor (CMOS).

In the color filter 32, for example, the filter elements of each of R,G, and B are arranged in a pattern called a Bayer array, as shown inFIG. 2. The Bayer array is an example of predetermined arrays of thefilter elements of R, G, and B. In FIG. 2, each of the filter elementsof G in each line held between the filter elements of R is indicated byGr, and each of the filter elements of G held between the filterelements of B is indicated by Gb.

The Bayer array has a configuration in which the horizontal linesalternating the filter elements of R with the filter elements of Gr andthe horizontal lines alternating the filter elements of B with thefilter elements of Gb are aligned alternately with each other in thevertical direction.

FIG. 3 shows spectral sensitive characteristics of wavelengths andrelative sensitivities of R light, G light, and B light in the imagingunit 3. The maximum value of the relative sensitivities is normalizedto 1. When the imaging device is operated in the normal mode, infraredlight having a wavelength of 700 nm or greater is required to be blockedin order to capture fine color images with visible light.

The drive unit 8 is thus controlled by a controller 7 to drive theoptical filter 2 in such a manner as to insert the infrared cut filter21 between the optical lens 1 and the imaging unit 3.

As is apparent from FIG. 3, the imaging unit 3 shows the sensitivitiesin the area where the infrared light having the wavelength of 700 nm orgreater is present. Therefore, when the imaging device is operated inthe intermediate mode or in the night-vision mode, the drive unit 8 iscontrolled by the controller 7 to drive the optical filter 2 in such amanner as to remove the infrared cut filter 21 from between the opticallens 1 and the imaging unit 3 and insert the dummy glass 22therebetween.

When the dummy glass 22 is inserted between the optical lens 1 and theimaging unit 3, the infrared light having the wavelength of 700 nm orgreater is not blocked. Thus, the imaging device can obtain informationof each of R, G and B by using the sensitivities in the oval regionsurrounded by the broken line in FIG. 3. The reason the dummy glass 22is inserted is to conform the optical path length obtained when thedummy glass 22 is used to the optical path length obtained when theinfrared cut filter 21 is used.

The infrared projector 9 includes projecting portions 91, 92, and 93 forprojecting infrared light with wavelengths IR1, IR2, and IR3,respectively. In the case of the intermediate mode or the night-visionmode, a projection controller 71 in the controller 7 controls theprojecting portions 91, 92, and 93 so as to selectively project theinfrared light with the respective wavelengths IR1, IR2, and IR3 in atime division manner.

A silicon wafer is used in the imaging element 31. FIG. 4 shows arelationship between wavelengths and relative detection rates when areflectance at each wavelength is multiplied by a light receivingsensitivity of silicon in a case where a material consisting of each ofthe colors R, G, and B is irradiated with white light. The maximum valueof the relative detection rates in FIG. 4 is also normalized to 1.

For example, as shown in FIG. 4, in the infrared light area, thereflected light with the wavelength of 780 nm has a strong correlationwith the reflected light of the material with color R, the reflectedlight with the wavelength of 870 nm has a strong correlation with thereflected light of the material with color B, and the reflected lightwith the wavelength of 940 nm has a strong correlation with thereflected light of the material with color G.

Thus, according to the present embodiment, the wavelengths IR1, IR2, andIR3 of infrared light projected from the projecting portions 91, 92, and93 are set to 780 nm, 940 nm, and 870 nm, respectively. These values areexamples for the wavelengths IR1, IR2, and IR3, and other wavelengthsother than 780 nm, 940 nm, and 870 nm may also be employed.

The projecting portion 91 radiates the infrared light with thewavelength IR1 on an object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to an Rsignal. The projecting portion 93 radiates the infrared light with thewavelength IR2 on the object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to a Gsignal. The projecting portion 92 radiates the infrared light with thewavelength IR3 on the object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to a Bsignal.

Accordingly, in the intermediate mode or in the night-vision mode, acolor similar to that obtained when the object is imaged in the normalmode in the state where visible light is present, can also be reproducedtheoretically.

Alternatively, the wavelength IR1 of 780 nm may be assigned to the Rlight, the wavelength IR3 of 870 nm may be assigned to the G light, andthe wavelength IR2 of 940 nm may be assigned to the B light, although inthis case the color image would possess a color tone different from theactual color tone of the object. The wavelengths IR1, IR2, and IR3 maybe assigned optionally to the R light, the G light, and the B light.

According to the present embodiment, the wavelengths IR1, IR2, and IR3are assigned to the R light, the G light, and the B light, respectively,by which the color tone of the object can be reproduced most finely.

The controller 7 controls the imaging unit 3 and the components includedin an image processing unit 5. An electronic shutter controller 73included in the controller 7 controls functions of an electronic shutterin the imaging unit 3. Image signals of images captured by the imagingunit 3 are subjected to A/D conversion by an A/D converter 4, and arethen input into the image processing unit 5. The imaging unit 3 and theA/D converter 4 may be integrated.

The controller 7 includes a mode switching unit 72 that switches betweenthe normal mode, the intermediate mod, and the night-vision mode. Themode switching unit 72 switches the operations in the image processingunit 5 as appropriate to correspond to the normal mode, the intermediatemode, and the night-vision mode, as described below. The imageprocessing unit 5 and the controller 7 may be integrated.

The image processing unit 5 includes switches 51 and 53, a pre-signalprocessing unit 52, and a demosaicing unit 54. The switches 51 and 53may be physical switches or may be logical switches for switching thepre-signal processing unit 52 between an active state and an inactivestate. The controller 7 receives an image signal output from the imageprocessing unit 5 in order to detect the brightness of the image beingcaptured.

As shown in FIG. 5, the pre-signal processing unit 52 includes asurrounding pixel adding unit 521, a same-position pixel adding unit522, and a synthesizing unit 523.

The image processing unit 5 generates data for the respective threeprimary colors R, G, and B, and supplies the data to the image outputunit 6. The image output unit 6 outputs the data for the three primarycolors in a predetermined format to a display unit (not shown) or thelike.

The image output unit 6 may directly output signals of the three primarycolors R, G and B, or may convert the signals of the three primarycolors R, G and B into luminance signals and color signals (or colordifference signals) before outputting. The image output unit 6 mayoutput composite image signals. The image output unit 6 may outputdigital image signals or output image signals converted into analogsignals by a D/A converter.

Next, the operations of each of the normal mode, the intermediate mode,and the night-vision mode are described in more detail below.

<Normal Mode>

In the normal mode, the controller 7 directs the drive unit 8 to insertthe infrared cut filter 21 between the optical lens 1 and the imagingunit 3. The projection controller 71 turns off the infrared projector 9to stop projecting infrared light.

Image signals captured by the imaging unit 3 are converted into imagedata as digital signals by the A/D converter 4, and then input into theimage processing unit 5. In the normal mode, the mode switching unit 72connects the switches 51 and 53 to the respective terminals Tb.

Item (a) of FIG. 6 shows exposures Ex1, Ex2, Ex 3, etc., of the imagingunit 3. Although the actual exposure time varies depending on conditionssuch as shutter speed, each of the exposures Ex1, Ex2, Ex 3, etc.,denotes the maximum exposure time. The shutter speed is determineddepending on the control by the electronic shutter controller 73.

Item (b) of FIG. 6 shows the timing at which each of frames of the imagesignals is obtained. Frame F0 of the image signals is obtained based onan exposure (not shown) prior to the exposure Ex1 after a predeterminedperiod of time. Frame F1 of the image signals is obtained based on theexposure Ex1 after a predetermined period of time. Frame F2 of the imagesignal is obtained based on the exposure Ex2 after a predeterminedperiod of time. The same operations are repeated after the exposure Ex3.A frame frequency of the image signals is, for example, 30 frames persecond.

The frame frequency of the image signals that may be determined asappropriate is that such as 30 frames per second or 60 frames per secondin the NTSC format, and 25 frames per second or 50 frames per second inthe PAL format. Alternatively, the frame frequency of the image signalsmay be 24 frames per second, which is used for movies.

The image data of each frame output from the A/D converter 4 is inputinto the demosaicing unit 54 via the switches 51 and 53. The demosaicingunit 54 subjects the image data of each input frame to demosaicing. Theimage processing unit 5 subjects the data to other types of imageprocessing in addition to the demosaicing, and outputs the data of thethree primary colors R, G and B.

The demosaicing in the demosaicing unit 54 is described below withreference to FIG. 7. Item (a) of FIG. 7 shows an arbitrary frame Fm ofimage data. The frame Fm is composed of pixels in an effective imageperiod. The number of the pixels is, for example, 640 horizontal pixelsand 480 vertical pixels in the VGA standard. For reasons ofsimplification, the number of the pixels in the frame Fm is greatlydecreased so as to schematically show the frame Fm.

The image data generated by the imaging unit 3 having the Bayer array isdata in which pixel data for R, G, and B are mixed in the frame Fm. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR, so as to generate interpolated pixel data Ri for R. The demosaicingunit 54 generates R frame FmR in which all pixels in one frame shown initem (b) of FIG. 7 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G, so as to generate interpolated pixel data Gi for G. Thedemosaicing unit 54 generates G frame FmG in which all pixels in oneframe shown in item (c) of FIG. 7 are composed of the pixel data for G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B, so as to generate interpolated pixel data Bi for B. Thedemosaicing unit 54 generates B frame FmB in which all pixels in oneframe shown in item (d) of FIG. 7 are composed of the pixel data for B.

The demosaicing unit 54 is only required to use at least the pixel datafor R when interpolating the pixel data for R, use at least the pixeldata for G when interpolating the pixel data for G, and use at least thepixel data for B when interpolating the pixel data for B. Alternatively,the demosaicing unit 54 may interpolate the pixel data for each of R, G,and B to be generated by use of the pixel data of the different colorsin order to improve the accuracy of the interpolation.

Since the imaging unit 3 further includes pixels outside the effectiveimage period, pixel data for each of R, G, and B can be interpolatedwith regard to the pixels located along the edges of top and bottom,left and right.

The R frame FmR, the G frame FmG and the B frame FmB generated by thedemosaicing unit 54 are output as the data for the three primary colorsR, G, and B. Although the pixel data for each of R, G, and B wasdescribed per frame in FIG. 7 for ease of explanation, the pixel datafor each of R, G, and B is actually output sequentially per pixel.

<Intermediate Mode: First Intermediate Mode>

In the intermediate mode (first intermediate mode and secondintermediate mode described below), the controller 7 directs the driveunit 8 to insert the dummy glass 22 between the optical lens 1 and theimaging unit 3. The projection controller 71 turns on the infraredprojector 9 to project infrared light. The mode switching unit 72connects the switches 51 and 53 to the respective terminals Ta.

Item (a) of FIG. 8 shows a state where infrared light is projected fromthe infrared projector 9. The controller 7 divides one frame period ofthe normal mode into three so as to control the projecting portions 91,92, and 93 to sequentially project infrared light in this order, forexample.

In the example of item (a) of FIG. 8, the infrared light with thewavelength IR1 (780 nm) is radiated on the object in the first ⅓ periodof the one frame. The infrared light with the wavelength IR2 (940 nm) isradiated on the object in the second ⅓ period of the one frame. Theinfrared light with the wavelength IR3 (870 nm) is radiated on theobject in the last ⅓ period of the one frame. The order of radiation ofthe infrared light with the respective wavelengths IR1, IR2, and IR3 isoptional.

As shown in item (b) of FIG. 8, exposure Ex1R which has a strongcorrelation with R light is executed by the imaging unit 3 at the pointwhere the infrared light with the wavelength IR1 is being projected.Exposure Ex1G which has a strong correlation with G light is executed bythe imaging unit 3 at the point where the infrared light with thewavelength IR2 is being projected. Exposure Ex1B which has a strongcorrelation with B light is executed by the imaging unit 3 at the pointwhere the infrared light with the wavelength IR3 is being projected.

Note that, since an image is captured in the intermediate mode in astate where visible light is slightly present, visible light andinfrared light projected from the infrared projector 9 coexist.Therefore, in the intermediate mode, exposures Ex1R, Ex1G, Ex1B, Ex2R,Ex2G, Ex2B, etc., are each obtained in a manner such that exposure ofvisible light and exposure of infrared light are combined together.

As shown in item (c) of FIG. 8, frame F1IR1 corresponding to theexposure Ex1R, frame F1IR2 corresponding to the exposure Ex1G, and frameF1IR3 corresponding to the exposure Ex1B are obtained based on theexposures Ex1R, Ex1G, and Ex1B after a predetermined period of time.

Further, frame F2IR1 corresponding to the exposure Ex2R, frame F2IR2corresponding to the exposure Ex2G, and frame F2IR3 corresponding to theexposure Ex2B are obtained based on the exposures Ex2R, Ex2G, and Ex2Bafter a predetermined period of time. The same operations are repeatedafter the exposures Ex3R, Ex3G, and Ex3B.

The frame frequency of the imaging signals in item (c) of FIG. 8 is 90frames per second. In the intermediate mode, one frame of the imagesignals in the normal mode is subjected to time division so as toproject the infrared light with the respective wavelengths IR1 to IR3.Thus, in order to output the image signals in the same format as thenormal mode, the frame frequency of the imaging signals in item (c) ofFIG. 8 is three times as many as that in the normal mode.

As described below, based on the imaging signals of the three frames initem (c) of FIG. 8, one frame of image signals is generated, having aframe frequency of 30 frames per second, as shown in item (d) of FIG. 8.For example, frame F1IR is generated based on the frames F1IR1, F1IR2,and F1IR3. Frame F2IR is generated based on the frames F2IR1, F2IR2, andF2IR3.

The operation of generating the image signals of each frame in item (d)of FIG. 8 in the intermediate mode, based on the imaging signals of thethree frames in item (c) of FIG. 8, is described in detail below.

The image data for the respective frames, corresponding to the imagingsignals shown in item (c) of FIG. 8 output from the A/D converter 4, isinput into the pre-signal processing unit 52 via the switch 51.

Pre-signal processing in the pre-signal processing unit 52 is describedbelow with reference to FIG. 9. Item (a) of FIG. 9 shows an arbitraryframe FmIR1 of image data generated at the point where the infraredlight with the wavelength IR1 is being projected. The pixel data foreach of R, B, Gr, and Gb in the frame FmIR1 is indicated with an index“1” indicating that all data is generated in the state where theinfrared light with the wavelength IR1 is projected.

Item (b) of FIG. 9 shows an arbitrary frame FmIR2 of image datagenerated at the point where the infrared light with the wavelength IR2is being projected. The pixel data for each of R, B, Gr, and Gb in theframe FmIR2 is indicated with an index “2” indicating that all data isgenerated in the state where the infrared light with the wavelength IR2is projected.

Item (c) of FIG. 9 shows an arbitrary frame FmIR3 of image datagenerated at the point where the infrared light with the wavelength IR3is being projected. The pixel data for each of R, B, Gr, and Gb in theframe FmIR3 is indicated with an index “3” indicating that all data isgenerated in the state where the infrared light with the wavelength IR3is projected.

Since the frame FmIR1 shown in item (a) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR1 having a strong correlation with R light is projected, the pixeldata for R is pixel data corresponding to the projected infrared light,and the pixel data for B and G are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof B, Gr, and Gb represents that the pixel data does not correspond tothe projected infrared light.

Since the frame FmIR2 shown in item (b) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR2 having a strong correlation with G light is projected, the pixeldata for G is pixel data corresponding to the projected infrared light,and the pixel data for R and B are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof R and B represents that the pixel data does not correspond to theprojected infrared light.

Since the frame FmIR3 shown in item (c) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR3 having a strong correlation with B light is projected, the pixeldata for B is pixel data corresponding to the projected infrared light,and the pixel data for R and G are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof R, Gr, and Gb represents that the pixel data does not correspond tothe projected infrared light.

The same-position pixel adding unit 522 in the pre-signal processingunit 52 individually adds the pixel data for each of R, Gr, Gb, and Blocated at the same pixel positions according to the following formulae(1) to (3) so as to generate added pixel data R123, Gr123, Gb123, andB123. In the intermediate mode, the surrounding pixel adding unit 521 inthe pre-signal processing unit 52 is inactive.R123=ka×R1+kb×R2+kc×R3  (1)G123=kd×G1+ke×G2+kf×G3  (2)B123=kg×B1+kh×B2+ki×B3  (3)In the formulae (1) to (3), R1, G1, and B1 are pixel data for R, G, andB in the frame FmIR1, R2, G2, and B2 are pixel data for R, G, and B inthe frame FmIR2, and R3, G3, and B3 are pixel data for R, G, and B inthe frame FmIR3. In addition, ka to ki are predetermined coefficients.The data G123 in the formula (2) is either Gr123 or Gb123.

The same-position pixel adding unit 522 adds the hatched pixel data foreach of R, Gr, Gb, and B to the pixel data for each of R, Gr, Gb, and Blocated at the same pixel positions not hatched.

In particular, the same-position pixel adding unit 522 adds, to thepixel data for R located in the frame FmIR1, the pixel data for Rlocated at the same pixel positions in each of the frames FmIR2 andFmIR3, so as to generate the added pixel data R123 according to theformula (1). That is, the same-position pixel adding unit 522 only usesthe pixel data in the region corresponding to the red color filter inthe light receiving elements and generates the added pixel data R123 forred.

The same-position pixel adding unit 522 adds, to the pixel data for Gr,Gb located in the frame FmIR2, the pixel data for Gr, Gb located at thesame pixel positions in each of the frames FmIR1 and FmIR3, so as togenerate the added pixel data G123 according to the formula (2). Thatis, the same-position pixel adding unit 522 only uses the pixel data inthe region corresponding to the green color filter in the lightreceiving elements and generates the added pixel data G123 for green.

The same-position pixel adding unit 522 adds, to the pixel data for Blocated in the frame FmIR3, the pixel data for B located at the samepixel positions in each of the frames FmIR1 and FmIR2, so as to generatethe added pixel data B123 according to the formula (3). That is, thesame-position pixel adding unit 522 only uses the pixel data in theregion corresponding to the blue color filter in the light receivingelements and generates the added pixel data B123 for blue.

The synthesizing unit 523 in the pre-signal processing unit 52 generatesframe FmIR123 of synthesized image signals shown in item (d) of FIG. 9based on the respective added pixel data R123, Gr123, Gb123, and B123generated at the respective pixel positions.

More particularly, the synthesizing unit 523 selects the added pixeldata R123 in the frame FmIR1, the added pixel data Gr123 and Gb123 inthe frame FmIR2, and the added pixel data B123 in FmIR3, and synthesizesthe respective added pixel data. The synthesizing unit 523 thusgenerates the frame FmIR123 of the synthesized image signals.

As described above, the synthesizing unit 523 generates the frameFmIR123 in which the respective added pixel data R123, Gr123, Gb123, andB123 are arranged so as to have the same array as the filter elements inthe color filter 32.

In the first intermediate mode, the image data in the frame FmIR123 aregenerated in such a manner as to use the pixel data not hatched and thepixel data hatched.

The reason the same-position pixel adding unit 522 adds the respectivepixel data located at the same pixel positions is that, since an imageis captured in the intermediate mode in the state where visible light ispresent, although the amount thereof is small, the hatched pixel datacontains the components of the respective colors based on the exposureby the visible light. Therefore, the respective pixel data located atthe same pixel positions are added to each other so that the sensitivityto the respective colors can be improved.

When the amount of visible light is relatively large in the state wherevisible light and infrared light coexist, the exposure by the visiblelight is predominant. In such a case, the image data in the frameFmIR123 mainly contains the components based on the image signalsexposed by the visible light. When the amount of infrared light isrelatively large in the state where infrared light and visible lightcoexist, the exposure by the infrared light is predominant. In such acase, the image data in the frame FmIR123 mainly contains the componentsbased on the image signals exposed by the infrared light.

When the amount of visible light is relatively small, the coefficientska, kb, and kc in the formula (1) preferably fulfill the relationship ofka>kb, kc, the coefficients kd, ke, and kf in the formula (2) preferablyfulfill the relationship of kf>kd, ke, and the coefficients kg, kh, andki in the formula (3) preferably fulfill the relationship of kh>kg, ki.This is because the wavelength IR1 has a strong correlation with the Rlight, the wavelength IR2 has a strong correlation with the G light, andthe wavelength IR3 has a strong correlation with the B light.

Accordingly, the pixel data for R can be the main data in the frameFmIR1, the pixel data for G can be the main data in the frame FmIR2, andthe pixel data for B can be the main data in the frame FmIR3.

The image data in the frame FmIR123 output from the pre-signalprocessing unit 52 is input into the demosaicing unit 54 via the switch53. The demosaicing unit 54 subjects the input image data in the frameFmIR123 to demosaicing in the same manner as the normal mode. The imageprocessing unit 5 subjects the image data to other types of imageprocessing in addition to the demosaicing, and outputs the data for thethree primary colors R, G, and B.

The demosaicing in the demosaicing unit 54 is described below withreference to FIG. 10. Item (a) of FIG. 10 shows the frame FmIR123. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR, so as to generate interpolated pixel data R123 i for R. Thedemosaicing unit 54 generates R frame FmIR123R in which all pixels inone frame shown in item (b) of FIG. 10 are composed of the pixel datafor R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G, so as to generate interpolated pixel data G123 i for G. Thedemosaicing unit 54 generates G frame FmIR123G in which all pixels inone frame shown in item (c) of FIG. 10 are composed of the pixel datafor G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B, so as to generate interpolated pixel data B123 i for B. Thedemosaicing unit 54 generates B frame FmIR123B in which all pixels inone frame shown in item (d) of FIG. 10 are composed of the pixel datafor B.

As is apparent from the operation of the demosaicing unit 54 in thenormal mode shown in FIG. 7 and the operation of the demosaicing unit 54in the intermediate mode shown in FIG. 10, the both operations aresubstantially the same. Thus, the operation of the demosaicing unit 54does not differ between the normal mode and the intermediate mode.

The pre-signal processing unit 52 is only required to be activated inthe intermediate mode except for the surrounding pixel adding unit 521,while the pre-signal processing unit 52 is inactivated in the normalmode. The normal mode and the intermediate mode may share the signalprocessing unit such as the demosaicing unit 54 in the image processingunit 5.

<Intermediate Mode: Second Intermediate Mode>

Operations in the second intermediate mode are described below withreference to FIG. 11 and FIG. 12. Note that the same operations as thosein the first intermediate mode are not repeated in the secondintermediate mode. The frame FmIR1, the frame FmIR2, and the frame FmIR3shown in items (a) to (c) in FIG. 11 are the same as the frame FmIR1,the frame FmIR2, and the frame FmIR3 shown in items (a) to (c) in FIG.9.

The synthesizing unit 523 selects pixel data R1 for R in the frameFmIR1, pixel data Gr2 and Gb2 for G in the frame FmIR2, and pixel dataB3 for B in FmIR3, and synthesizes the respective pixel data. Thesynthesizing unit 523 thus generates frame FmIR123′ of the synthesizedimage signals shown in item (d) of FIG. 11.

That is, the frame FmIR123′ is image data in which the pixel data for R,Gr, Gb, and B not hatched in each of the frames FmIR1, FmIR2, and FmIR3are collected in one frame.

Thus, the frame FmIR123′ contains the pixel data for red only using thepixel data in the region corresponding to the red color filter in thestate where the infrared light with the wavelength IR1 is projected, thepixel data for green only using the pixel data in the regioncorresponding to the green color filter in the state where the infraredlight with the wavelength IR2 is projected, and the pixel data for blueonly using the pixel data in the region corresponding to the blue colorfilter in the state where the infrared light with the wavelength IR3 isprojected.

As described above, the synthesizing unit 523 generates the frameFmIR123′ in which the respective pixel data R1, Gr2, Gb2, and B3 arearranged so as to have the same array as the filter elements in thecolor filter 32.

In the second intermediate mode, the same-position pixel adding unit 522defines the coefficient ka in the formula (1) as 1 and the othercoefficients kb and kc as 0, defines the coefficient ke in the formula(2) as 1 and the other coefficients kd and kf as 0, and defines thecoefficient ki in the formula (3) as 1 and the other coefficients kg andkh as 0.

Therefore, the value of the pixel data for R in the frame FmIR1, thevalues of the pixel data for Gr and Gb in the frame FmIR2, and the valueof the pixel data for B in the frame FmIR3 each remain as is.

Accordingly, the synthesizing unit 523 can generate the frame FmIR123′by selecting the pixel data for R in the frame FmIR1, the pixel data forGr and Gb in the frame FmIR2, and the pixel data for B in the frameFmIR3, in the same manner as the operations in the first intermediatemode.

In the second intermediate mode, the pre-signal processing unit 52 onlyuses the pixel data (the pixel data not hatched) generated in the statewhere the infrared light for generating the pixel data with the samecolor is projected so as to generate the frame FmIR123′.

According to the second intermediate mode, although the sensitivity orcolor reproduction performance decreases compared with the firstintermediate mode, the calculation processing can be simplified or theframe memory can be reduced.

The demosaicing in the demosaicing unit 54 is described below withreference to FIG. 12. Item (a) of FIG. 12 shows the frame FmIR123′. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR, so as to generate interpolated pixel data R1 i for R. The demosaicingunit 54 generates R frame FmIR123′R in which all pixels in one frameshown in item (b) of FIG. 12 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G, so as to generate interpolated pixel data G2 i for G. Thedemosaicing unit 54 generates G frame FmIR123′G in which all pixels inone frame shown in item (c) of FIG. 12 are composed of the pixel datafor G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B, so as to generate interpolated pixel data B3 i for B. Thedemosaicing unit 54 generates B frame FmIR123′B in which all pixels inone frame shown in item (d) of FIG. 12 are composed of the pixel datafor B.

Accordingly, in the intermediate mode, the pixel data for red isgenerated from the pixel data obtained from the region corresponding tothe red color filter in the light receiving elements, the pixel data forgreen is generated from the pixel data obtained from the regioncorresponding to the green color filter in the light receiving elements,and the pixel data for blue is generated from the pixel data obtainedfrom the region corresponding to the blue color filter in the lightreceiving elements.

<Night-Vision Mode: First Night-Vision Mode>

In the night-vision mode (first night-vision mode and secondnight-vision mode described below), the controller 7 directs the driveunit 8 to insert the dummy glass 22 between the optical lens 1 and theimaging unit 3, as in the case of the intermediate mode. The projectioncontroller 71 turns on the infrared projector 9 to project infraredlight. The mode switching unit 72 connects the switches 51 and 53 to therespective terminals Ta.

The general operations in the night-vision mode are the same as thoseshown in FIG. 8. However, since an image is captured in the night-visionmode in a state where almost no visible light is present, the exposuresEx1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B, etc., shown in item (b) of FIG. 8are assumed to be exposure only by infrared light.

Under the condition that there is almost no visible light but onlyinfrared light, the characteristics of the respective filter elements inthe color filter 32 do not differ from each other. Thus, the imagingunit 3 can be considered as a single-color imaging device.

Therefore, in the night-vision mode, the surrounding pixel adding unit521 in the pre-signal processing unit 52 adds surrounding pixel data toall pixel data in order to improve the sensitivity of infrared light.

More particularly, when the R pixel is the target pixel as shown in item(a) of FIG. 13, the surrounding pixel adding unit 521 adds, to the pixeldata for R as the target pixel, the pixel data of the surrounding eightpixels of G (Gr, Gb) and B.

While the pixel data for red is generated from the pixel data obtainedfrom the region corresponding to the red color filter in the lightreceiving elements in the intermediate mode, the pixel data for red isgenerated, in the night-vision mode, from the pixel data obtained from awider region than the region in the intermediate mode. The respectiveexamples shown in items (a) to (d) of FIG. 13 use the pixel dataobtained from the region of the nine pixels including the target pixel.

When the Gr pixel is the target pixel as shown in item (b) of FIG. 13,the surrounding pixel adding unit 521 adds, to the pixel data for Gr asthe target pixel, the pixel data of the surrounding eight pixels of R,Gb, and B. When the Gb pixel is the target pixel as shown in item (c) ofFIG. 13, the surrounding pixel adding unit 521 adds, to the pixel datafor Gb as the target pixel, the pixel data of the surrounding eightpixels of R, Gr, and B.

While the pixel data for green is generated from the pixel data obtainedfrom the region corresponding to the green color filter in the lightreceiving elements in the intermediate mode, the pixel data for green isgenerated, in the night-vision mode, from the pixel data obtained from awider region than the region in the intermediate mode.

When the B pixel is a target pixel as shown in item (d) of FIG. 13, thesurrounding pixel adding unit 521 adds, to the pixel data for B as thetarget pixel, the pixel data of the surrounding eight pixels of R and G.

While the pixel data for blue is generated from the pixel data obtainedfrom the region corresponding to the blue color filter in the lightreceiving elements in the intermediate mode, the pixel data for blue isgenerated, in the night-vision mode, from the pixel data obtained from awider region than the region in the intermediate mode.

The surrounding pixel adding unit 521 may simply add the pixel data ofthe nine pixels together including the target pixel and the surroundingeight pixels, or may add, to the pixel data of the target pixel, thepixel data of the surrounding eight pixels after being subjected toparticular weighting processing.

There is a known imaging element capable of collectively reading out aplurality of pixels as a single pixel, which is called binning. When theimaging element possessing the binning function is used as the imagingelement 31, the adding processing may be performed not by thesurrounding pixel adding unit 521 but by the imaging element with thisbinning function. The binning processing performed by the imagingelement is substantially equivalent to the adding processing performedby the surrounding pixel adding unit 521.

The frames FmIR1, FmIR2, and FmIR3 shown in items (a) to (c) of FIG. 14are the same as the frames FmIR1, FmIR2, and FmIR3 shown in items (a) to(c) of FIG. 9, respectively. In items (d) to (f) of FIG. 14, each ofadded pixel data R1 ad, Gr1 ad, Gb1 ad, B1 ad, R2 ad, Gr2 ad, Gb2 ad, B2ad, R3 ad, Gr3 ad, Gb3 ad, and B3 ad is obtained in a manner such thatthe pixel data of the surrounding eight pixels are added to the pixeldata for each of R, Gr, Gb, and B.

The surrounding pixel adding unit 521 subjects the pixel data in each ofthe frames FmIR1, FmIR2, and FmIR3 to adding processing shown in FIG.13, so as to generate frame FmIR1 ad, frame FmIR2 ad, and frame FmIR3 adshown in items (d) to (f) of FIG. 14.

The frames FmIR1 ad, FmIR2 ad, and FmIR3 ad shown in items (a) to (c) ofFIG. 15 are the same as the frames FmIR1 ad, FmIR2 ad, and FmIR3 adshown in items (d) to (f) of FIG. 14, respectively.

As in the case of the first intermediate mode, the same-position pixeladding unit 522 adds, to the pixel data R1 ad located in the frame FmIR1ad, the pixel data R2 ad and R3 ad located at the same pixel positionsin the respective frames FmIR2 ad and FmIR3 ad, so as to generate addedpixel data R123 ad according to the formula (1).

The same-position pixel adding unit 522 adds, to the pixel data Gr2 adand Gb2 ad located in the frame FmIR2 ad, the pixel data Gr1 ad, Gb1 ad,Gr3 ad, and Gb3 ad located at the same pixel positions in the respectiveframes FmIR1 ad and FmIR3 ad, so as to generate added pixel data Gr123ad and Gb123 ad according to the formula (2).

The same-position pixel adding unit 522 adds, to the pixel data B3 adlocated in the frame FmIR3 ad, the pixel data B1 ad and B2 ad located atthe same pixel positions in the respective frames FmIR1 ad and FmIR2 ad,so as to generate added pixel data B123 ad according to the formula (3).

As in the case of the first intermediate mode, the synthesizing unit 523selects the added pixel data R123 ad in the frame FmIR1 ad, the addedpixel data Gr123 ad and Gb123 ad in the frame FmIR2 ad, and the addedpixel data B123 ad in FmIR3 ad, and synthesizes the respective addedpixel data. The synthesizing unit 523 thus generates frame FmIR123 ad ofthe synthesized image signals shown in item (d) of FIG. 15.

The synthesizing unit 523 generates the frame FmIR123 ad in which therespective added pixel data R123 ad, Gr123 ad, Gb123 ad, and B123 ad arearranged so as to have the same array as the filter elements in thecolor filter 32.

Item (a) of FIG. 16 shows the frame FmIR123 ad. The demosaicing unit 54computes pixel data for R for pixel positions where no pixel data for Ris present by use of the surrounding pixel data for R, so as to generateinterpolated pixel data R123 adi for R. The demosaicing unit 54generates R frame FmIR123 adR in which all pixels in one frame shown initem (b) of FIG. 16 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G, so as to generate interpolated pixel data G123 adi for G.The demosaicing unit 54 generates G frame FmIR123 adG in which allpixels in one frame shown in item (c) of FIG. 16 are composed of thepixel data for G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B, so as to generate interpolated pixel data B123 adi for B.The demosaicing unit 54 generates B frame FmIR123 adB in which allpixels in one frame shown in item (d) of FIG. 16 are composed of thepixel data for B.

The first intermediate mode and the first night-vision mode differ fromeach other in that the surrounding pixel adding unit 521 is inactive inthe first intermediate mode, and the surrounding pixel adding unit 521is active in the first night-vision mode. The mode switching unit 72 isonly required to activate the surrounding pixel adding unit 521 when inthe night-vision mode.

The operation of the demosaicing unit 54 in the night-vision mode issubstantially the same as that in the normal mode and in theintermediate mode. The normal mode, the intermediate mode, and thenight-vision mode may share the signal processing unit such as thedemosaicing unit 54 in the image processing unit 5.

<Night-Vision Mode: Second Night-Vision Mode>

Operations in the second night-vision mode are described below withreference to FIG. 17 and FIG. 18. Note that the same operations as thosein the first night-vision mode are not described in the secondnight-vision mode. The frames FmIR1 ad, FmIR2 ad, and FmIR3 ad shown initems (a) to (c) in FIG. 17 are the same as the frames FmIR1 ad, FmIR2ad, and FmIR3 ad shown in items (a) to (c) in FIG. 15.

The synthesizing unit 523 selects pixel data R1 ad for R in the frameFmIR1 ad, pixel data Gr2 ad and Gb2 ad for G in the frame FmIR2 ad, andpixel data B3 ad for B in FmIR3 ad and synthesizes the respective pixeldata. The synthesizing unit 523 thus generates frame FmIR123′ ad of thesynthesized image signals shown in item (d) of FIG. 17.

The synthesizing unit 523 generates the frame FmIR123′ ad in which therespective pixel data R1 ad, Gr2 ad, Gb2 ad, and B3 ad are arranged soas to have the same array as the filter elements in the color filter 32.

As described with reference to FIG. 13, the pixel data R1 ad for red inthe frame FmIR123′ ad is generated from the pixel data obtained from awider region than the region used for generating the pixel data for redwhen in the intermediate mode.

The pixel data Gr2 ad for green in the frame FmIR123′ ad is generatedfrom the pixel data obtained from a wider region than the region usedfor generating the pixel data for green when in the intermediate mode.

The pixel data B3 ad for blue in the frame FmIR123′ ad is generated fromthe pixel data obtained from a wider region than the region used forgenerating the pixel data for blue when in the intermediate mode.

As in the case of the second intermediate mode, the same-position pixeladding unit 522 in the second night-vision mode defines the coefficientka in the formula (1) as 1 and the other coefficients kb and kc as 0,defines the coefficient ke in the formula (2) as 1 and the othercoefficients kd and kf as 0, and defines the coefficient ki in theformula (3) as 1 and the other coefficients kg and kh as 0.

Therefore, the value of the pixel data R1 ad in the frame FmIR1 ad, thevalues of the pixel data Gr2 ad and Gb2 ad in the frame FmIR2 ad, andthe value of the pixel data B3 ad in the frame FmIR3 ad each remain asis.

Accordingly, the synthesizing unit 523 can generate the frame FmIR123′ad by selecting the pixel data R1 ad in the frame FmIR1 ad, the pixeldata Gr2 ad and Gb2 ad in the frame FmIR2 ad, and the pixel data B3 adin the frame FmIR3 ad, in the same manner as the operations in the firstnight-vision mode.

The demosaicing in the demosaicing unit 54 is described below withreference to FIG. 18. Item (a) of FIG. 18 shows the frame FmIR123′ ad.The demosaicing unit 54 computes pixel data for R for pixel positionswhere no pixel data for R is present by use of the surrounding pixeldata R1 ad, so as to generate interpolated pixel data R1 adi for R. Thedemosaicing unit 54 generates R frame FmIR123′adR in which all pixels inone frame shown in item (b) of FIG. 18 are composed of the pixel datafor R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata Gr2 ad and Gb2 ad, so as to generate interpolated pixel data G2 adifor G. The demosaicing unit 54 generates G frame FmIR123′adG in whichall pixels in one frame shown in item (c) of FIG. 18 are composed of thepixel data for G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata B3 ad, so as to generate interpolated pixel data B3 adi for B. Thedemosaicing unit 54 generates B frame FmIR123′adB in which all pixels inone frame shown in item (d) of FIG. 18 are composed of the pixel datafor B.

The second intermediate mode and the second night-vision mode differfrom each other in that the surrounding pixel adding unit 521 isinactive in the second intermediate mode, and the surrounding pixeladding unit 521 is active in the second night-vision mode.

While the pixel data for each color is generated from the pixel dataobtained from the region corresponding to each color filter in the lightreceiving elements in the intermediate mode, the pixel data for eachcolor is generated, in the night-vision mode, from the pixel dataobtained from a wider region than the region used for generating thepixel data for each color in the intermediate mode, as the surroundingpixels are added in the night-vision mode.

<Example of Mode Switch>

An example of mode switching by the mode switching unit 72 is describedbelow with reference to FIG. 19. Item (a) of FIG. 19 is an exampleschematically showing a state of change in environmental brightness withthe passage of time from daytime to nighttime.

As shown in item (a) of FIG. 19, the brightness gradually decreases withthe passage of time from daytime to nighttime, and results in almosttotal darkness after time t3. Item (a) of FIG. 19 shows the brightnessrepresenting a substantial amount of visible light, and indicates thatalmost no visible light is present after time t3.

The controller 7 can determine the environmental brightness based on abrightness level of image signals (image data) input from the imageprocessing unit 5. As shown item (b) of FIG. 19, the mode switching unit72 selects the normal mode when the brightness is predeterminedthreshold Th1 (first threshold) or greater, selects the intermediatemode when the brightness is less than the threshold Th1 andpredetermined threshold Th2 (second threshold) or greater, and selectsthe night-vision mode when the brightness is less than the thresholdTh2.

The imaging device according to the present embodiment automaticallyswitches the modes in such a manner as to select the normal mode by timet1 at which the brightness reaches the threshold Th1, select theintermediate mode in the period from time t1 to time t2 at which thebrightness reaches the threshold Th2, and select the night-vision modeafter time t2. In item (b) of FIG. 19, the intermediate mode may beeither the first intermediate mode or the second intermediate mode, andthe night-vision mode may be either the first night-vision mode or thesecond night-vision mode.

Although the brightness immediately before time t3 at which almost novisible light remains is defined as the threshold Th2 in item (a) ofFIG. 19, the brightness at time t3 may be defined as the threshold Th2.

As shown in item (c) of FIG. 19, the mode switching unit 72 may dividethe intermediate mode into two periods: a first half period toward timet1 as the first intermediate mode in which the amount of visible lightis relatively high; and a second half period toward time t2 as thesecond intermediate mode in which the amount of visible light isrelatively low. In item (c) of FIG. 19, the night-vision mode may beeither the first night-vision mode or the second night-vision mode.

In the imaging device according to the present embodiment, theprojection controller 71 controls the ON/OFF state of the infraredprojector 9, and the mode switching unit 72 switches the respectivemembers in the image processing unit 5 between the active state and theinactive state, so as to implement the respective modes.

As shown in FIG. 20, the normal mode is a state where the infraredprojector 9 is turned OFF, the surrounding pixel adding unit 521, thesame-position pixel adding unit 522, and the synthesizing unit 523 areinactive, and the demosaicing unit 54 is active.

The first intermediate mode is implemented in a state where the infraredprojector 9 is turned ON, the surrounding pixel adding unit 521 isinactive, and the same-position pixel adding unit 522, the synthesizingunit 523, and the demosaicing unit 54 are active. The secondintermediate mode is implemented in a state where the infrared projector9 is turned ON, the surrounding pixel adding unit 521 and thesame-position pixel adding unit 522 are inactive, and the synthesizingunit 523 and the demosaicing unit 54 are active.

The same-position pixel adding unit 522 can be easily switched betweenthe active state and the inactive state by appropriately setting thecoefficients ka to ki in the formulae (1) to (3), as described above.

The first night-vision mode is implemented in a state where the infraredprojector 9 is turned ON, and the surrounding pixel adding unit 521, thesame-position pixel adding unit 522, the synthesizing unit 523, and thedemosaicing unit 54 are all active. The second night-vision mode isimplemented in a state where the infrared projector 9 is turned ON, thesame-position pixel adding unit 522 is inactive, and the surroundingpixel adding unit 521, the synthesizing unit 523, and the demosaicingunit 54 are active.

The surrounding pixel adding unit 521 can be activated in the processingof adding the surrounding pixels by setting the coefficient to greaterthan 0 (for example, 1) by which the surrounding pixel data ismultiplied in the calculation formula used for adding the surroundingpixel data to the pixel data of the target pixel.

The surrounding pixel adding unit 521 can be inactivated in theprocessing of adding the surrounding pixels by setting the coefficientto 0 by which the surrounding pixel data is multiplied in thecalculation formula.

The surrounding pixel adding unit 521 thus can easily be switchedbetween the active state and the inactive state by setting thecoefficient as appropriate.

First Modified Example of Imaging Device

The method of detecting the environmental brightness by the controller 7is not limited to the method based on the brightness level of the imagesignals.

As shown in FIG. 21, the environmental brightness may be detected by abrightness sensor 11. In FIG. 21, the environmental brightness may bedetermined based on both the brightness level of the image signals andthe environmental brightness detected by the brightness sensor 11.

Second Modified Example of Imaging Device

The controller 7 may briefly estimate the environmental brightness basedon the season (date) and the time (time zone) during a year, instead ofthe direct detection of the environmental brightness, so as to switchthe modes by the mode switching unit 72.

As shown in FIG. 22, the normal mode, the intermediate mode, and thenight-vision mode are set in a mode setting table 12 depending on thecombination of the date and the time zone. A time clock 73 in thecontroller 7 manages the date and the time. The controller 7 refers tothe date and the time indicated on the time clock 73 so as to read outthe mode set in the mode setting table 12.

The projection controller 71 and the mode switching unit 72 control theimaging device so as to select the mode read from the mode setting table12.

Third Modified Example of Imaging Device

As shown in FIG. 23, a user may control the imaging device with anoperation unit 13 by manually selecting one of the modes, so as to setthe projection controller 71 and the mode switching unit 72 to theselected mode. The operation unit 13 may be operated using the operationbuttons provided on the casing of the imaging device or by a remotecontroller.

<Image Signal Processing Method>

The image signal processing method executed by the imaging device shownin FIG. 1 is again described with reference to FIG. 24.

In FIG. 24, once the imaging device starts operating, the controller 7determines in step S1 whether the environmental brightness is thethreshold Th1 or greater. When the environmental brightness is thethreshold Th1 or greater (YES), the controller 7 executes the processingin the normal mode in step S3. When the environmental brightness is notthe threshold Th1 or greater (NO), the controller 7 determines in stepS2 whether the environmental brightness is threshold Th2 or greater.

When the environmental brightness is the threshold Th2 or greater (YES),the controller 7 executes the processing in the intermediate mode instep S4. When the environmental brightness is not the threshold Th2 orgreater (NO), the controller 7 executes the processing in thenight-vision mode in step S5.

The controller 7 returns the processing to step S1 after executing theprocessing from steps S3 to S5, and repeats the respective followingsteps.

FIG. 25 shows the specific processing in the normal mode in step S3. InFIG. 25, the controller 7 (the projection controller 71) turns off theinfrared projector 9 in step S31. The controller 7 inserts the infraredcut filter 21 in step S32. The controller 7 (the mode switching unit 72)connects the switches 51 and 53 to the respective terminals Tb in stepS33. The execution order from steps S31 to S33 is optional. The stepsS31 to S33 can be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS34. The controller 7 controls the image processing unit 5 in step S35so that the demosaicing unit 54 subjects, to demosaicing, a framecomposing image signals generated when the imaging unit 3 images theobject.

FIG. 26 shows the specific processing in the intermediate mode in stepS4. In FIG. 26, the controller 7 (the projection controller 71) turns onthe infrared projector 9 in step S41 so that the projecting portions 91to 93 project infrared light with the respective wavelengths IR1 to IR3in a time division manner.

The controller 7 inserts the dummy glass 22 in step S42. The controller7 (the mode switching unit 72) connects the switches 51 and 53 to therespective terminals Ta in step S43. The execution order from steps S41to S43 is optional. The steps S41 to S43 may be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS44. The imaging unit 3 images the object in a state where the infraredlight with the wavelength IR1 assigned to R, the infrared light with thewavelength IR2 assigned to G, and the infrared light with the wavelengthIR3 assigned to B, are each projected.

The controller 7 (the mode switching unit 72) controls the pre-signalprocessing unit 52 in step S45 so as to inactivate the surrounding pixeladding unit 521 and activate the synthesizing unit 523 to generatesynthesized image signals.

The respective frames composing the image signals generated when theimaging unit 3 images the object in the state where the infrared lightwith the respective wavelengths IR1, IR2, and IR3 is projected, aredefined as a first frame, a second frame, and a third frame.

The synthesizing unit 523 arranges the pixel data for the three primarycolors based on the pixel data for R in the first frame, the pixel datafor G in the second frame, and the pixel data for B in the third frame,so as to have the same array as the filter elements in the color filter32. The synthesizing unit 523 thus generates the synthesized imagesignals in a manner such that the image signals in the first to thirdframes are synthesized in one frame.

The controller 7 controls the image processing unit 5 in step S46 sothat the demosaicing unit 54 subjects the frame composing thesynthesized image signals to demosaicing.

The demosaicing unit 54 executes, based on the frame of the synthesizedimage signals, demosaicing for generating an R frame, a G frame, and a Bframe, so as to sequentially generate the frames of the three primarycolors subjected to demosaicing.

The demosaicing unit 54 can generate the R frame by interpolating thepixel data for R in the pixel positions where no pixel data for R ispresent. The demosaicing unit 54 can generate the G frame byinterpolating the pixel data for G in the pixel positions where no pixeldata for G is present. The demosaicing unit 54 can generate the B frameby interpolating the pixel data for B in the pixel positions where nopixel data for B is present.

When executing the operations in the first intermediate mode, thecontroller 7 activates the same-position pixel adding unit 522 in stepS45. When executing the operations in the second intermediate mode, thecontroller 7 inactivates the same-position pixel adding unit 522 in stepS45.

FIG. 27 shows the specific processing in the night-vision mode in stepS5. In FIG. 27, the controller 7 (the projection controller 71) turns onthe infrared projector 9 in step S51 so that the projecting portions 91to 93 project infrared light with the respective wavelengths IR1 to IR3in a time division manner.

The controller 7 inserts the dummy glass 22 in step S52. The controller7 (the mode switching unit 72) connects the switches 51 and 53 to therespective terminals Ta in step S53. The execution order from steps S51to S53 is optional. The steps S51 to S53 may be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS54. The controller 7 (the mode switching unit 72) controls thepre-signal processing unit 52 in step S55 so as to activate thesurrounding pixel adding unit 521 and the synthesizing unit 523 togenerate synthesized image signals.

The controller 7 controls the image processing unit 5 in step S56 sothat the demosaicing unit 54 subjects the frame composing thesynthesized image signals to demosaicing.

When executing the operations in the first night-vision mode, thecontroller 7 activates the same-position pixel adding unit 522 in stepS55. When executing the operations in the second night-vision mode, thecontroller 7 inactivates the same-position pixel adding unit 522 in stepS55.

<Image Signal Processing Program>

In FIG. 1, the controller 7 or the integrated portion of the imageprocessing unit 5 and the controller 7 may be composed of a computer(microcomputer), and an image signal processing program (computerprogram) may be executed by the computer, so as to implement the sameoperations as those in the imaging device described above.

An example of a procedure of the processing executed by the computerwhen the processing in the intermediate mode executed in step S4 shownin FIG. 24 is included in the image signal processing program, isdescribed below with reference to FIG. 28. FIG. 28 shows the processingexecuted by the computer instructed by the image signal processingprogram.

In FIG. 28, the image signal processing program instructs the computerto control the infrared projector 9 in step S401 to project infraredlight with the wavelengths IR1, IR2, and IR3 assigned to R, G, and B,respectively.

The step in step S401 may be executed by an external unit outside of theimage signal processing program. In FIG. 28, the step of inserting thedummy glass 22 is omitted. The step of inserting the dummy glass 22 maybe executed by the external unit outside of the image signal processingprogram.

The image signal processing program instructs the computer in step S402to obtain the pixel data composing the first frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR1 is projected.

The image signal processing program instructs the computer in step S403to obtain the pixel data composing the second frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR2 is projected.

The image signal processing program instructs the computer in step S404to obtain the pixel data composing the third frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR3 is projected. The executionorder from steps S402 to 404 is optional.

The image signal processing program instructs the computer in step S405to arrange the respective pixel data for R, G, and B in such a manner asto have the same array as the filter elements in the color filter 32, soas to generate the synthesized image signals synthesized in one frame.

In the intermediate mode, the image signal processing program does notinstruct the computer to execute the processing of adding thesurrounding pixels in step S405.

The image signal processing program instructs the computer in step S406to subject the frame of the synthesized image signals to demosaicing, soas to generate the frames of R, G, and B.

Although not illustrated in the drawing, the image signal processingprogram may instruct the computer to execute the processing of addingthe surrounding pixels in step S405 shown in FIG. 28 when the processingin the night-vision mode executed in step S5 shown in FIG. 24 isincluded in the image signal processing program.

The image signal processing program may be a computer program stored ina storage medium readable on the computer. The image signal processingprogram may be provided in a state of being stored in the storagemedium, or may be provided via a network such as the Internet in amanner such that the image signal processing program is downloaded tothe computer. The storage medium readable on the computer may be anarbitrary non-transitory storage medium, such as CD-ROM and DVD-ROM.

<Reduction in Color Variation when Imaging Moving Object>

Next, variations in color and a method of reducing the variations incolor are described below, the variations in color being caused when theimaging device according to the present embodiment images a movingobject in the intermediate mode or the night-vision mode as describedabove.

FIG. 29 is a view schematically showing a method for controlling theimaging device when the imaging device generates a frame of an imagesignal, while taking no account of variations in color.

Item (a) of FIG. 29 is the same as item (a) of FIG. 8, showing a statewhere infrared light is projected from the infrared projector 9. FIG. 29shows a case where the period in which the infrared light of each of thewavelengths IR1 to IR3 is projected is not the whole one frame period ofthe maximum exposure time, but is shorter than the one frame period.

As shown in item (b) of FIG. 29, the exposures Ex1R, Ex1G, Ex1B, Ex2R,Ex2G, Ex2B, etc., are each obtained at the point when the infrared lightof each of the wavelengths IR1 to IR3 is projected.

Item (c) of FIG. 29 shows frames F of images imaged by the exposuresshown in item (b) of FIG. 29. FIG. 29 only shows the frames F obtainedby the exposures Ex1R, Ex1G, and Ex1B. As shown in the three frames F,the rectangular object OB is assumed to be moving at speed v from leftto right in the horizontal direction.

As shown in item (d) of FIG. 29, the frames F1IR1, F1IR2, and F1IR3 areobtained based on the exposures Ex1R, Ex1G, and Ex1B. The frames F2IR1,F2IR2, and F2IR3 are obtained based on the exposures Ex2R, Ex2G, andEx2B. The frames F3IR1, F3IR2, and F3IR3 are obtained based on theexposures Ex3R, Ex3G, and Ex3B.

The imaging signals shown in item (b) and the frames shown in item (d)each have a frame frequency of 90 frames per second. The set ofexposures Ex1R, Ex1G and Ex1B, the set of exposures Ex2R, Ex2G and Ex2B,and the set of exposures Ex3R, Ex3G and Ex3B each have a frame frequencyof 30 frames per second.

As shown in item (e) of FIG. 29, the frames F1IR, F1IR2, and F1IR3 aresynthesized to generate the frame F1IR. The frames F2IR, F2IR2, andF2IR3 are synthesized to generate the frame F2IR. The frames R1IR andF2IR each have a frame frequency of 30 frames per second.

As shown in items (a) and (b) of FIG. 29, the middle point of the periodin which the infrared light of each of the wavelengths IR1 to IR3 isprojected corresponds to the middle point of the one frame period of themaximum exposure time.

Since the object OB is irradiated with the infrared light of thewavelength IR1 for generating the R signal during the exposure Ex1R, theobject OB in the frame F1IR1 is indicated by red or an equivalent color.Since the object OB is irradiated with the infrared light of thewavelength IR2 for generating the G signal during the exposure Ex1G, theobject OB in the frame F1IR2 is indicated by green or an equivalentcolor.

Since the object OB is irradiated with the infrared light of thewavelength IR3 for generating the B signal during the exposure Ex1B, theobject OB in the frame F1IR3 is indicated by blue or an equivalentcolor.

In actual cases, sometimes the object OB cannot be indicated by therespective corresponding colors depending on the material of the objectOB; however, for reasons of expediency, it is assumed that the object OBin the frame F1IR1 is indicated by red, the object OB in the frame F1IR2is indicated by green, and the object OB in the frame F1IR3 is indicatedby blue.

The interval between the middle points of the respective maximumexposure times shown in item (b) of FIG. 29 is defined as time t0. Theobject OB moves a distance [ΔLrg=v×t0] between the frames F1IR1 andF1IR2. Therefore, as shown in item (f) of FIG. 29, the frame F1IR isprovided with the color-shift region C1 indicated by red, and having alength corresponding to the distance ΔLrg.

The frame F1IR, adjacent to the color-shift region C1, is provided withthe color-shift region C2 indicated by yellow caused such that theobject OB in red in the frame F1IR1 is superimposed on the object OB ingreen in the frame F1IR2. Note that the color-shift region C2 does notnecessarily result in actual yellow, but is assumed to have yellow forreasons of expediency.

The frame F1IR, adjacent to the color-shift region C2, is provided withthe region C3 indicated by a proper color, obtained such that theobjects OB in the respective frames F1IR1, F1IR2, and F1IR3 are allsuperimposed together.

The frame F1IR, adjacent to the region C3, is provided with thecolor-shift region C4 indicated by cyan caused such that the object OBin green in the frame F1IR2 is superimposed on the object OB in blue inthe frame F1IR3. Note that the color-shift region C4 does notnecessarily result in actual cyan, but is assumed to have cyan forreasons of expediency.

The object OB moves by a distance [ΔLgb=v×t0] between the frames F1IR2and F1IR3. Therefore, the frame F1IR, adjacent to the color-shift regionC4, is provided with the color-shift region C5 indicated by blue andhaving a length corresponding to the distance ΔLgb.

As described above, when the object OB is a moving object, thecolor-shift regions C1, C2, C4, and C5 are caused around the region C3indicated by the proper color, which leads to a deterioration of theimage quality.

FIGS. 30 to 32 show the preferred method for controlling the imagingdevice in order to minimize variations in color when imaging the objectOB, which is moving. The respective controlling methods shown in FIGS.30 to 32 are described below in order.

Item (b) of FIG. 30 shows exposure-start timing ExRs and exposure-endtiming ExRe for the exposure Ex1R, exposure-start timing ExGs andexposure-end timing ExGe for the exposure Ex1G, and exposure-starttiming ExBs and exposure-end timing ExBe for the exposure Ex1B.

FIG. 30 shows a first example of the controlling method. As shown initem (a) of FIG. 30, the middle point of the period in which theinfrared light of the wavelength IR1 is projected is shifted forwardtoward the exposure-end timing ExRe, from the middle point of themaximum exposure time of the exposure Ex1R.

The middle point of the period in which the infrared light of thewavelength IR3 is projected is shifted backward toward theexposure-start timing ExBs from the middle point of the maximum exposuretime of the exposure Ex1B.

The middle point of the period in which the infrared light of thewavelength IR2 is projected corresponds to the middle point of themaximum exposure time of the exposure Ex1G, as in the case of item (a)of FIG. 29.

The middle point of the period in which the infrared light of thewavelength IR2 is projected does not necessarily correspond to themiddle point of the maximum exposure time of the exposure Ex1G; however,these middle points preferably correspond to each other.

The timing of projecting the infrared light of the wavelengths IR1 toIR3 with respect to the exposures Ex2R, Ex2G, and Ex2B, the exposuresEx3R, Ex3G, and Ex3B, etc., is the same as the timing of projecting theinfrared light of the wavelengths IR1 to IR3 with respect to theexposures Ex1R, Ex1G, and Ex1B.

The interval between the middle point of the period in which theinfrared light of the wavelength IR2 is projected, and the middle pointof the period in which the infrared light of the wavelength IR1 isprojected, is defined as time t1. The interval between the middle pointof the period in which the infrared light of the wavelength IR2 isprojected, and the middle point of the period in which the infraredlight of the wavelength IR3 is projected, is defined as time t1.

The time t1 is shorter than the time t0 by Δt. Therefore, the distancethat the object OB moves between the frames F1IR1 and F1IR2 is reducedby ΔL=v×Δt. The distance that the object OB moves between the framesF1IR2 and F1IR3 is also reduced by ΔL=v×Δt.

Accordingly, as shown in item (f) of FIG. 30, each length of thecolor-shift regions C1, C2, C4, and C5 is reduced by ΔL. This increasesthe length of the region C3 indicated by the proper color and leads to areduction in color variation.

In the first example, the middle point of the period in which theinfrared light of the wavelength IR1 is projected is shifted toward theexposure-end timing ExRe, and the middle point of the period in whichthe infrared light of the wavelength IR3 is projected is shifted towardthe exposure-start timing ExBs.

When only the middle point of the period in which the infrared light ofthe wavelength IR1 is projected is shifted toward the exposure-endtiming ExRe, the color-shift regions C1 and C2 can be decreased. Whenonly the middle point of the period in which the infrared light of thewavelength IR3 is projected is shifted toward the exposure-start timingExBs, the color-shift regions C4 and C5 can be decreased.

In the first example, as described above, the interval between themiddle point of the period of the infrared light projected in themiddle, and the middle point of the period of the infrared lightprojected before or after the middle infrared light, is shorter than theinterval between the middle point of the period of the infrared lightprojected in the middle, and the middle point of the maximum exposuretime of the exposure Ex1R or Ex1B. Thus, the first example can minimizevariations in color.

The configurations of the imaging device controlled by the controllingmethod of the first example are summarized as follows. The firstinfrared light, the second infrared light, and the third infrared lightare sequentially projected. The first infrared light has the firstwavelength assigned to the first color of red, green, and blue. Thesecond infrared light has the second wavelength assigned to the secondcolor of red, green, and blue. The third infrared light has the thirdwavelength assigned to the third color of red, green, and blue.

The projection controller 71 controls the infrared projector 9 tosequentially project the first infrared light, the second infraredlight, and the third infrared light.

The imaging unit 3 images an object in a state where the first infraredlight is projected in at least part of one frame period so as togenerate the first frame based on the first imaging signal. The oneframe period is determined depending on the maximum exposure time in theimaging unit 3.

The imaging unit 3 images the object in a state where the secondinfrared light is projected in at least part of the one frame period soas to generate the second frame based on the second imaging signal. Theimaging unit 3 images the object in a state where the third infraredlight is projected in at least part of the one frame period so as togenerate the third frame based on the third imaging signal.

The image processing unit 5 synthesizes the first to third frames togenerate a frame of an image signal.

The middle point of the period in which the second infrared light isprojected is defined as the first timing. The middle point of the periodin which the first or third infrared light is projected is defined asthe second timing. The middle point of the one frame period of the firstor third frame is defined as the third timing.

The projection controller 71 sets the interval between the first timingand the second timing shorter than the interval between the first timingand the third timing, and controls the infrared projector 9 to projectthe first to third infrared lights.

The middle point of the one frame period of the second frame is definedas the fourth timing. It is particularly preferable that the projectioncontroller 71 control the infrared projector 9 to project the secondinfrared light by conforming the first timing to the fourth timing.

A control program (computer program) of the imaging device may beexecuted by a computer so as to implement the operations of the imagingdevice controlled by the controlling method of the first example asdescribed above. The control program of the imaging device may be acomputer program stored in a non-transitory storage medium readable on acomputer, as in the case of the image signal processing programdescribed above.

More particularly, the control program of the imaging device is executedby the computer to implement the first step of controlling the infraredprojector 9 to project the first infrared light and the second step ofgenerating the first frame. The control program of the imaging device isexecuted by the computer to implement the third step of controlling theinfrared projector 9 to project the second infrared light and the fourthstep of generating the second frame.

The control program of the imaging device is executed by the computer toimplement the fifth step of controlling the infrared projector 9 toproject the third infrared light, the sixth step of generating the thirdframe, and the seventh step of synthesizing the first to third frames togenerate a frame of an image signal.

The control program of the imaging device sets the interval between thefirst timing and the second timing shorter than the interval between thefirst timing and the third timing.

A second example of the controlling method shown in FIG. 31 is describedbelow, mainly with regard to the differences between this example andthe first example shown in FIG. 30. Items (a) and (c) to (g) of FIG. 31are the same as items (a) to (f) of FIG. 30.

In the second example, as shown in item (a) of FIG. 31, the infraredlight having each of the wavelengths IR1 to IR3 is projectedapproximately during the whole one frame period of the respectiveexposures Ex1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B, Ex3R, Ex3G, Ex3B, etc.

Item (b) of FIG. 31 shows the period and timing in which the electronicshutter of the imaging unit 3 is released, according to the control bythe electronic shutter controller 73. In the frame period of theexposure Ex1G, the middle point of electronic shutter-release periodSt12 corresponds to the middle point of the maximum exposure time of theexposure Ex1G. In the frame period of the exposure Ex1R, the middlepoint of electronic shutter-release period St11 is shifted forwardtoward the exposure-end timing ExRe from the middle point of the maximumexposure time of the exposure Ex1R. In the frame period of the exposureEx1B, the middle point of electronic shutter-release period St13 isshifted backward toward the exposure-start timing ExBs from the middlepoint of the maximum exposure time of the exposure Ex1B.

The timing of the electronic shutter-release periods St21, St22, andSt23, the electronic shutter-release periods St31, St32, and St33, etc.,with respect to the exposures Ex2R, Ex2G, and Ex2B, the exposures Ex3R,Ex3G, and Ex3B, etc., respectively, is the same as the timing of theelectronic shutter-release periods St11, St12, and St13 with respect tothe exposures Ex1R, Ex1G, and Ex1B.

Even when the infrared light is projected during the whole one frameperiod of each exposure, the imaging signal obtained by imaging theobject OB irradiated with the infrared light is input into the A/Dconverter only for the electronic shutter-release period.

The second example can therefore obtain the frame F1IR, including theregion C3 indicated by the proper color and the color-shift regions C1,C2, C4, and C5, as shown in item (g) of FIG. 31, as in the case of theframe F1IR shown in item (f) of FIG. 30. Accordingly, the second examplecan also minimize variations in color.

A third example of the controlling method shown in FIG. 32 is describedbelow, mainly with regard to the differences between this example, thefirst example shown in FIG. 30, and the second example shown in FIG. 31.Items (a) to (g) of FIG. 32 correspond to items (a) to (g) of FIG. 31,respectively.

In the third example, as shown in items (a) and (b) of FIG. 32, theperiod in which the infrared light with the wavelengths IR1 to IR3 isprojected corresponds to each electronic shutter-release period. Theperiod and timing in which the infrared light having the respectivewavelengths IR1 to IR3 is projected in the third example are the same asthose in the first example shown in FIG. 30.

The third example differs from the first example in that each electronicshutter-release period corresponds to the period in which the infraredlight is projected. The third example can also minimize variations incolor.

The configurations of the imaging device controlled by the controllingmethod of the second or third example are summarized as follows, whichare different from those of the first example.

The electronic shutter controller 73 controls the functions of theelectronic shutter in the imaging unit 3. The middle point of the periodin which the imaging unit 3 is exposed while the second infrared lightis projected is defined as the first timing. The middle point of theperiod in which the imaging unit 3 is exposed while the first or thirdinfrared light is projected is defined as the second timing. The middlepoint of the one frame period of the first or third frame is defined asthe third timing.

The electronic shutter controller 73 controls the period and timing inwhich the imaging unit 3 is exposed such that the interval between thefirst timing and the second timing is set shorter than the intervalbetween the first timing and the third timing.

The control program (computer program) of the imaging device may beexecuted by the computer so as to implement the operations of theimaging device controlled by the controlling method of the second orthird example as described above.

The control program of the imaging device can therefore be executed bythe computer, by use of the functions of the electronic shutter in theimaging unit 3, to implement processing to control the period and timingin which the imaging unit 3 is exposed such that the interval betweenthe first timing and the second timing is set shorter than the intervalbetween the first timing and the third timing.

The present invention is not limited to the embodiments described above,and various modifications and improvements can be made without departingfrom the scope of the present invention. The controller 7 and the imageprocessing unit 5 may be composed of one or more hardware components(circuits or processors). The use of hardware or software is optional.The imaging device may only include hardware, or part of the imagingdevice may be composed of software.

What is claimed is:
 1. An imaging device comprising: a projectioncontroller configured to control an infrared projector to selectivelyand sequentially project, in the following order, a first infrared lighthaving a first wavelength assigned to a first color of red, green, andblue, a second infrared light having a second wavelength assigned to asecond color of red, green, and blue, and a third infrared light havinga third wavelength assigned to a third color of red, green, and blue,wherein the first, second and third wavelengths are each at least 700nm; an imaging unit configured to image an object in a state where thefirst infrared light is projected in at least part of one frame periodso as to generate a first frame based on a first imaging signal, toimage the object in a state where the second infrared light is projectedin at least part of the one frame period so as to generate a secondframe based on a second imaging signal, and to image the object in astate where the third infrared light is projected in at least part ofthe one frame period so as to generate a third frame based on a thirdimaging signal, wherein the first, second and third frames are generatedin this order; and an image processing unit configured to synthesize thefirst to third frames to generate a frame of an image signal, whereinthe projection controller sets an interval between a first timing and asecond timing shorter than an interval between the first timing and athird timing, the first timing being a middle point of a period in whichthe second infrared light is projected, the second timing being a middlepoint of a period in which the first or third infrared light isprojected, and the third timing being a middle point of the one frameperiod of the first or third frame, and controls the infrared projectorto project the first to third infrared lights, and wherein when thesecond timing is the middle point of the period in which the imagingunit is exposed in the state where the first infrared light isprojected, the third timing is the middle point of the one frame periodof the first frame, and when the second timing is the middle point ofthe period in which the imaging unit is exposed in the state where thethird infrared light is projected, the third timing is the middle pointof the one frame period of third frame.
 2. The imaging device accordingto claim 1, wherein the projection controller controls the infraredprojector to project the second infrared light by conforming the firsttiming to a fourth timing that is a middle point of the one frame periodof the second frame.
 3. A method for controlling an imaging device,comprising: a first step of imaging an object by an imaging unit in astate where a first infrared light having a first wavelength assigned toa first color of red, green, and blue is projected in at least part ofone frame period so as to generate a first frame based on a firstimaging signal, wherein the first wavelength is at least 700 nm; asecond step, implemented after the first step, of imaging the object bythe imaging unit in a state where a second infrared light having asecond wavelength assigned to a second color of red, green, and blue isprojected in at least part of the one frame period so as to generate asecond frame based on a second imaging signal, wherein the secondwavelength is at least 700 nm; a third step, implemented after thesecond step, of imaging the object by the imaging unit in a state wherea third infrared light having a third wavelength assigned to a thirdcolor of red, green, and blue is projected in at least part of the oneframe period so as to generate a third frame based on a third imagingsignal, wherein the third wavelength is at least 700 nm; and a fourthstep of synthesizing the first to third frames to generate a frame of animage signal, wherein an interval between a first timing and a secondtiming is set shorter than an interval between the first timing and athird timing, the first timing being a middle point of a period in whichthe second infrared light is projected, the second timing being a middlepoint of a period in which the first or third infrared light isprojected, and the third timing being a middle point of the one frameperiod of the first or third frame, and wherein when the second timingis the middle point of the period in which the imaging unit is exposedin the state where the first infrared light is projected, the thirdtiming is the middle point of the one frame period of the first frame,and when the second timing is the middle point of the period in whichthe imaging unit is exposed in the state where the third infrared lightis projected, the third timing is the middle point of the one frameperiod of third frame.
 4. A computer software product that includes anon-transitory storage medium readable by a processor, thenon-transitory storage medium having stored thereon a set ofinstructions for performing control of an imaging device theinstructions comprising: (a) a first set of instructions which, whenloaded into main memory and executed by the processor, causes theprocessor to control, by a projection controller, an infrared projectorto project a first infrared light having a first wavelength assigned toa first color of red, green, and blue wherein the first wavelength is atleast 700 nm; (b) a second set of instructions which, when loaded intomain memory and executed by the processor, causes the processor to imagean object by an imaging unit in a state where the first infrared lightis projected in at least part of one frame period so as to generate afirst frame based on a first imaging signal; (c) a third set ofinstructions which, when loaded into main memory and executed by theprocessor, causes the processor to continue from the first set ofinstructions to control the infrared projector to project a secondinfrared light having a second wavelength assigned to a second color ofred, green, and blue, wherein the second wavelength is at least 700 nm;(d) a fourth set of instructions which, when loaded into main memory andexecuted by the processor, causes the processor to image the object bythe imaging unit in a state where the second infrared light is projectedin at least part of the one frame period so as to generate a secondframe based on a second imaging signal; (e) a fifth set of instructionswhich, when loaded into main memory and executed by the processor,causes the processor to continue from the third set of instructions tocontrol the infrared projector to project a third infrared light havinga third wavelength assigned to a third color of red, green, and blue,wherein the third wavelength is at least 700 nm; (f) a sixth set ofinstructions which, when loaded into main memory and executed by theprocessor, causes the processor to image the object by the imaging unitin a state where the third infrared light is projected in at least partof the one frame period so as to generate a third frame based on a thirdimaging signal; and (g) a seventh set of instructions which, when loadedinto main memory and executed by the processor, causes the processor tosynthesize the first to third frames to generate a frame of an imagesignal using an image processing unit, wherein an interval is setbetween a first timing and a second timing shorter than an intervalbetween the first timing and a third timing, the first timing being amiddle point of a period in which the second infrared light isprojected, the second timing being a middle point of a period in whichthe first or third infrared light is projected, and the third timingbeing a middle point of the one frame period of the first or thirdframe, and wherein when the second timing is the middle point of theperiod in which the imaging unit is exposed in the state where the firstinfrared light is projected, the third timing is the middle point of theone frame period of the first frame, and when the second timing is themiddle point of the period in which the imaging unit is exposed in thestate where the third infrared light is projected, the third timing isthe middle point of the one frame period of third frame.